Search for: All records

Algorithmic bias research often evaluates models in terms of traditional demographic categories (e.g., U.S. Census), but these categories may not capture nuanced, context-dependent identities relevant to learning. This study evaluates four affect detectors (boredom, confusion, engaged concentration, and frustration) developed for an adaptive math learning system. Metrics for algorithmic fairness (AUC, weighted F1, MADD) show subgroup differences across several categories that emerged from a free-response social identity survey (Twenty Statements Test; TST), including both those that mirror demographic categories (i.e., race and gender) as well as novel categories (i.e., Learner Identity, Interpersonal Style, and Sense of Competence). For demographic categories, the confusion detector performs better for boys than for girls and underperforms for West African students. Among novel categories, biases are found related to learner identity (boredom, engaged concentration, and confusion) and interpersonal style (confusion), but not for sense of competence. Results highlight the importance of using contextually grounded social identities to evaluate bias. 

This study investigates stopout patterns in MOOCs to understand course and assessment-level factors that influence student stopout behavior. We expanded previous work on stopout by assessing the exponential decay of assessment-level stopout rates across courses. Results confirm a disproportionate stopout rate on the first graded assessment. We then evaluated which course and assessment level features were associated with stopout on the first assessment. Findings suggest that a higher number of questions and estimated time commitment in the early assessments and more assessments in a course may be associated with a higher proportion of early stopout behavior. 

Students' reading ability affects their outcomes in learning software even outside of reading education, such as in math education, which can result in unexpected and inequitable outcomes. We analyze an adaptive learning software using Bayesian Knowledge Tracing (BKT) to understand how the fairness of the software is impacted when reading ability is not modeled. We tested BKT model fairness by comparing two years of data from 8,549 students who were classified as either "emerging" or "non-emerging" readers (i.e., a measure of reading ability). We found that while BKT was unbiased on average in terms of equal predictive accuracy across groups, specific skills within the adaptive learning software exhibited bias related to reading level. Additionally, there were differences between the first-answer mastery rates of the emerging and non-emerging readers (M=.687 and M=.776, difference CI=[0.075, 0.095]), indicating that emerging reader status is predictive of mastery. Our findings demonstrate significant group differences in BKT models regarding reading ability, exhibiting that it is important to consider—and perhaps even model—reading as a separate skill that differentially influences students' outcomes."]} 

While MOOCs have been widely studied in terms of student engagement and academic performance, the extent to which engagement within MOOCs predict career advancement remains underexplored. Building on prior work, this study investigates how participation in discussion forums, specifically social presence and the use of course-relevant keywords, affects career advancement. Using GPT-assisted content analysis of forum posts, we assess how these engagement factors relate to both achievement during the course and post-course career advancement. Our findings indicate that social presence and use of course-relevant keywords has a positive relationship with course achievement during the MOOC. However, no significant relationship was found between career advancement and either social presence or course-related keywords in discussion forums. These findings suggest that while active engagement in MOOC discussion forums enhances academic achievement, it might not directly translate into career advancement, highlighting a possible disconnect between learning participation in MOOCS and professional outcomes. 

This study explores the potential of the large language model GPT-4 as an automated tool for qualitative data analysis by educational researchers, exploring which techniques are most successful for different types of constructs. Specifically, we assess three different prompt engineering strategies — Zero-shot, Few-shot, and Few-shot with contextual information — as well as the use of embeddings. We do so in the context of qualitatively coding three distinct educational datasets: Algebra I semi-personalized tutoring session transcripts, student observations in a game-based learning environment, and debugging behaviours in an introductory programming course. We evaluated the performance of each approach based on its inter-rater agreement with human coders and explored how different methods vary in effectiveness depending on a construct’s degree of clarity, concreteness, objectivity, granularity, and specificity. Our findings suggest that while GPT-4 can code a broad range of constructs, no single method consistently outperforms the others, and the selection of a particular method should be tailored to the specific properties of the construct and context being analyzed. We also found that GPT-4 has the most difficulty with the same constructs than human coders find more difficult to reach inter-rater reliability on. 

There has been increasing interest in data enclaves in recent years, both in education and other fields. Data enclaves make it possible to conduct analysis on large-scale and higher-risk data sets, while protecting the privacy of the individuals whose data is included in the data sets, thus mitigating risks around data disclosure. In this article, we provide a post-mortem on the MORF (MOoc Replication Framework) 2.1 infrastructure, a data enclave expected to sunset and be replaced in the upcoming years, reviewing the core factors that reduced its usefulness for the community. We discuss challenges to researchers in terms of usability, including challenges involving learning to use core technologies, working with data that cannot be directly viewed, debugging, and working with restricted outputs. Our post-mortem discusses possibilities for ways that future infrastructures could get past these challenges. 

Adaptive learning systems are increasingly common in U.S. classrooms, but it is not yet clear whether their positive impacts are realized equally across all students. This study explores whether nuanced identity categories from open-ended self-reported data are associated with outcomes in an adaptive learning system for secondary mathematics. As a measure of impact of these social identity data, we correlate student responses for 3 categories: race and ethnicity, gender, and learning identity—a category combining student status and orientation toward learning—and total lessons completed in an adaptive learning system over one academic year. Results show the value of emergent and novel identity categories when measuring student outcomes, as learning identity was positively correlated with mathematics outcomes across two statistical tests. 

Research on epistemic emotions has often focused on how students transition between affective states (e.g., affect dynamics). More recently, studies have examined the properties of cases where a student remains in the same affective state over time, finding that the duration of a student's affective state is important for multiple learning outcomes. However, the likelihood of remaining in a given affective state has not been widely studied across different methods or systems. Additionally, the role of motivational factors in the persistence or decay of affective states remains underexplored. This study builds on two prior investigations into the exponential decay of epistemic emotions, expanding the analysis of affective chronometry by incorporating two detection methods based on student self-reports and trained observer labels in a game-based learning environment. We also examine the relationship between motivational measures and affective decay. Our findings indicate that boredom exhibits the slowest decay across both detection methods, while confusion is the least persistent. Furthermore, we found that higher situational interest and self-efficacy are associated with greater persistence in engaged concentration, as identified by both detection methods. This work provides novel insights into how motivational factors shape affective chronometry, contributing to a deeper understanding of the temporal dynamics of epistemic emotions. 

Students in open-ended educational games have a number of different pathways that they can select to work productively through a learning activity. Educators and system designers may want to know which of these pathways are most effective for engagement, learning, or other desirable outcomes. In this paper, we investigate which prior jobs and factors are associated with higher rates of student quitting behavior in an educational science exploration game. We use a series of Chi squared analyses to identify the jobs with the highest rates of quitting overall, and we calculate logistic regressions within specific jobs to determine the potential factors that lead to students quitting those jobs. Our analysis revealed that for 23 of the 40 jobs examined, having experience in at least one previous job significantly decreased the chances of students quitting the subsequent job, and that completing specific prior jobs reduces quit rates on specific later jobs. In our discussion, we describe the challenges associated with modeling quitting behavior, and how these analyses could be used to better optimize students’ pathways through the game environment. Specially, guiding students through specific sequences of preliminary jobs before tackling more challenging jobs can improve their engagement and reduce dropout rates, thus optimizing their learning pathways.